Weighted Aggregating Stochastic Gradient Descent for Parallel Deep Learning

This paper investigates the stochastic optimization problem focusing on developing scalable parallel algorithms for deep learning tasks. Our solution involves a reformation of objective function in neural network models, along with novel computing strategy, coined weighted aggregating gradient descent ( WASGD ). Following theoretical analysis characteristics new function, introduces decentralized scheme based performance local workers. Without any center variable, method automatically gauges importance workers and accepts them by their contributions. Furthermore, we have developed an enhanced version method, xmlns:xlink="http://www.w3.org/1999/xlink">WASGD+ , (1) implementing designed sample order (2) upgrading weight evaluation function. To validate benchmark our pipeline against several popular including state-of-the-art classifier training techniques (e.g., elastic averaging SGD). Comprehensive validation studies been conducted four classic datasets: xmlns:xlink="http://www.w3.org/1999/xlink">CIFAR-100 xmlns:xlink="http://www.w3.org/1999/xlink">CIFAR-10 xmlns:xlink="http://www.w3.org/1999/xlink">Fashion-MNIST xmlns:xlink="http://www.w3.org/1999/xlink">MNIST . Subsequent results firmly validated superiority accelerating architecture. Better still, version, is shown to be significant improvement over its prototype.